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CLAIMS 

1 . Method for generating persistent annotations of multimedia content, comprising one or 
more repetitions of the following steps: 

actively selecting examples of multimedia content to be annotated by a user; 
accepting input annotations from said user for said selected examples; 
propagating said input annotations to other instances of multimedia content; and 
storing said input annotations and said propagated annotations, 

2. The method of claim 1, wherein the step of actively selecting is performed using a 
selection technique selected from the group consisting of: deterministic and probabilistic. 

3. The method of claim 2, wherein the step of actively selecting, which is performed 
deterministically or probabilistically, is based on explicit models and feature proximity/similarity 
measures, and returns one or more examples of multimedia content to be annotated. 

4. The method of claim 2, wherein the step of actively selecting, which is performed 
deterministically or probabilistically, is based on implicit models and feature proximity/similarity 
measures, and returns one or more examples of multimedia content to be annotated. 

5. The method of claim 1, wherein an optimization criterion for active selection includes 
one or more criteria selected from the group consisting of: maximizing disambiguation, 
information measures, and confidence. 

6. The method of claim 1, wherein the multimedia content comprises one or more types 
selected from the group consisting of: images, audio, video, graphics, text, multimedia, Web 
pages, time series data, surveillance data, sensor data, relational data, and XML data. 

7. The method of claim 1 , wherein the input annotations are created by a user with reference 
to a vocabulary. 
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8. The method of claim 7, wherein the vocabulary contains one or more items selected from 
the group consisting of: terms, concepts, labels, and annotations, 

9. The method of claim 1 , wherein the process of creating input annotations by the user 
involves multimodal interaction with the user using graphical, textual, and/or speech interface. 

5 10. The method of claim 1, wherein the input annotations are created by means of steps 
selected from the group consisting of: creating new annotations, deleting existing annotations, 
rejecting proposed annotations, and modifying annotations. 

1 1 . The method of claim 7, wherein the vocabulary is adaptively or dynamically organized 
and/or limited by the system or the user. 

10 12. The method of claim 9, wherein the multimodal interaction involves speech recognition, 
gaze detection, finger pointing, expression detection, and/or effective computing methods for 
sensing a user's state. 



|J 13. The method of claim 1 , wherein the determination of the propagation of annotations is made 
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5 J deterministically or probabilistically and on the use of models for each annotation or for joint 

015 annotations. 
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14. The method of Claim 2, wherein the models are created or learned automatically or 
semi-automatically and/or are updated adaptively from interaction with the user, 

15. The method of claim 2, wherein the models are based on nearest neighbor voting or variants, 
parametric or statistical models, expert systems, rule-based systems, or hybrid techniques. 

20 16. System for generating persistent annotations of multimedia content, comprising: 

means for actively selecting examples of multimedia content to be annotated by a user; 
means for accepting input annotations from said user for said selected examples; 
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means for propagating said input annotations to other instances of multimedia content; 
and 

means for storing said input annotations and said propagated annotations. 

17. The system of claim 16 wherein the means for actively selecting uses a selection 
5 technique selected from the group consisting of: deterministic and probabilistic. 

18. The system of claim 17, wherein the means for actively selecting, which uses a 
deterministic or probabilistic technique, is based on explicit models and feature 
proximity/similarity measures, and returns one or more examples of multimedia content to be 
annotated. 

0 10 19. The system of claim 17, wherein the means for actively selecting, which uses a 

Iff deterministic or probabilistic technique, is based on implicit models and feature 
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jjjjjj. proximity/similarity measures, and returns one or more examples of multimedia content to be 

Tj annotated. 
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20. The system of claim 16, wherein an optimization criterion for active selection includes 
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f |j 1 5 one or more criteria selected from the group consisting of: maximizing disambiguation, 

0 information measures, and confidence. 
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21 . The system of claim 1 6, wherein the multimedia content comprises one or more types 
selected from the group consisting of: images, audio, video, graphics, text, multimedia, Web 
pages, time series data, surveillance data, sensor data, relational data, and XML data. 

20 22. A computer program product in a computer readable medium for generating persistent 
annotations of multimedia content, the computer program product comprising instructions for 
performing one or more repetitions of the following steps: 

actively selecting of examples of multimedia content to be annotated by a user; 

accepting input annotations from said user for said selected examples; 
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propagating said input annotations to other instances of multimedia content; and 
storing said input annotations and said propagated annotations. 
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